Light field image sensor, method and applications

ABSTRACT

An angle-sensitive pixel (ASP) device that uses the Talbot effect to detect the local intensity and incident angle of light includes a phase grating disposed above a photodiode assembly or a phase grating disposed above an analyzer grating that is disposed above a photodiode assembly. When illuminated by a plane wave, the upper grating generates a self-image at a selected Talbot depth. Several such structures, tuned to different incident angles, are sufficient to extract local incident angle and intensity. Arrays of such structures are sufficient to localize light sources in three dimensions without any additional optics.

RELATED APPLICATION DATA

The instant application is a continuation-in-part of and claims priorityto U.S. application Ser. No. 13/055,566 filed on Apr. 6, 2011, which isa US National Stage filing of PCT/US09/51660 filed on Jul. 24, 2009,which claims priority to U.S. Provisional application Ser. No.61/083,688 filed on Jul. 25, 2008, and further claims priority to U.S.Provisional application Ser. No. 61/407,202 filed on Oct. 27, 2010, thesubject matters of which are incorporated herein by reference in theirentireties.

FEDERALLY SPONSORED RESEARCH

This invention was made with Government support under Grant Number5R21EB009841-01 awarded by the National Institute of Health. The UnitedStates Government has certain rights in the invention.

BACKGROUND

1. Field of the Invention

Embodiments of the invention are generally directed to the field oflight field sensing and light field image detection. More particularly,embodiments of the invention are directed to a lens-less,angle-sensitive pixel (ASP) sensor and ASP devices having increasedquantum efficiency and pixel density, which can measure the intensityand incident angle of a light field to provide an image of the lightfield. Embodiments of the invention further include, without limitation,imaging methods associated with said sensor and device embodiments, andapplications thereof.

2. Related Art Discussion

Conventional imaging uses a large array of light sensors to create a mapof light intensity at an image plane. However, this intensity map failsto capture incident angle, polarization angle, and other properties oflight rays passing through the image plane. A complete description ofthese additional parameters defines the light field or, “flow” of light,at the image plane.

Michael Faraday first proposed the concept of light as a field in themid 1800's. This concept was expanded by the theory of a “light field”in three-dimensional space. At a given point, the light field is definedby the infinite collection of vectors that represent the light arrivingat the point from all angles. The light field can be formally defined bya “plenoptic function” of multiple variables. The plenoptic functionparameterizes the light rays passing through all space in terms ofintensity, I, which is dependent on position in space (x, y, z),direction (θ, φ), wavelength (λ), time (t), and polarization angle (p).Hence, I(x, y, z, θ, φ, λ, t, p) is the complete representation of avisual scene and contains all possible views of the scene.

Measuring the plenoptic function would require an observer to be able todetermine the intensity of every ray, for every wavelength, at allinstants in time and at every point in space. Clearly, perfectdetermination of the plenoptic function for any practical scene isimpossible. However, a number of techniques collectively known aslight-field imaging have been devised that can record aspects of theplenoptic function beyond simple intensity at a plane. One reportedmethod is to use an array of pinhole cameras, where each camera capturesthe incident angle-dependent intensity I(θ, φ) at a particular location,(x₀, y₀). Cameras at different positions (x_(i), y_(i)) capture a sliceof the plenoptic function, I(x, y, θ, φ). Arrays of conventional camerascan also be used, as can camera scanning, or multiple masks. Small-scalesolutions have used micro-lenses to emulate camera arrays. However, allof these approaches require a significant number of parallel or moveableoptical components to capture information about the light field beyond asimple intensity map.

Recording information about the light field of a scene provides a morecomplete description of that scene than a conventional photograph ormovie, and is useful for a number of applications. The light fieldallows prediction of illumination patterns on a surface given knownsources and the three-dimensional reconstruction of scenes (e.g.,“light-field rendering” or “three-dimensional shape approximation”).FIGS. 1 a, 1 b show how one aspect of the light field, e.g., incidentangle, can be used to localize a light source in three-dimensionalspace. Capturing the light field also permits construction of imageswith an arbitrary focal plane and aperture. This capability is useful inboth photography and in microscopy for obtaining multiple focal planeswithout moving optics.

A wide variety of applications require information about thethree-dimensional structure of microscale samples. Direct capture ofthis information using commodity semiconductor chips with no additionalhardware would reduce the size and cost of many instruments and assaysby orders of magnitude. Existing solid-state image sensors employ largearrays of photosensitive pixels that capture an intensity map ofincident light, but no angle information. In typical imagingapplications, a lens is required to ensure that the intensity maprepresents some object of interest. Without a lens, one must rely purelyon the information contained in the light rays striking the imagesensor. If a sample is placed sufficiently close to the image sensor andilluminated, the resulting intensity map will typically contain someblurred two-dimensional spatial information. Three-dimensionalinformation is completely lost. Information contained in the incidentangle of light rays is of interest because it contains furtherrecoverable spatial information.

To date, macroscopic angle-detectors have been demonstrated inunmodified integrated circuit technology. Pixel-scale angle-sensitivestructures have been demonstrated on chip but require post-assembledarrays of microlenses, which significantly increase cost and complexityover the manufacture and use of standard imagers.

Another reported technique involves silicon-on-insulator (SOI)structures utilizing regions of metal that are large relative to thewavelength of the light to generate a shadow on an underlyingphotodiode. This approach has been reportedly used to perform a singleangle measurement but is not well suited to deployment in imager arrays.

The inventors recognize that solutions and improvements to theshortcomings and challenges in the prior art are necessary and would bebeneficial. More specifically, in contrast to other approaches thatrequire multiple lenses and/or moving parts, devices that aremonolithic, require no optical components aside from the sensor itself,and which can be manufactured in a standard planar microfabricationprocess (e.g., CMOS) would be advantageous in the art. The embodimentsof the invention disclosed and claimed herein successfully address thesematters and achieve these goals.

The inventors further recognize that the metallic structures used toform the micron-scale fine-pitch transmission amplitude gratings tocreate the interference patterns from the incident light field, of theinstant invention, block a significant fraction of the available light.While reduced light sensitivity is not a significant problem for manyapplications, maintaining high sensitivity comparable to that of atraditional photodetector permits more widespread deployment ofangle-sensitive imagers. In addition, the combination of this ‘top’grating and an ‘analyzer’ grating as described herein, results in astructure of relatively significant size as well as sub-optimal quantumefficiency (QE). It would also be beneficial to improve angular acuityand reduce the wavelength dependence of the previously embodied ASPs.Accordingly there is a need for an improved ASP apparatus and associatedsystems and methods that address these problems and concerns withoutcompromising basic function or CMOS manufacturing capability.

SUMMARY

Embodiments of the invention are directed to apparatus and methods formeasuring a light field at a given image plane. Pixel and detectordevices disclosed herein are sensitive to both the intensity and theincident angle of incident light from an object scene. The disclosedapparatus and methods utilize the Talbot effect of periodic lightdiffracting structures to characterize incident light by its magnitudeand direction. In certain aspects, local, micron-scale diffractiongratings at each of a large number of sensor sites are used to capturethis information. To distinguish certain of these devices from thetypical pixels of digital image sensors, we refer to them herein as“angle-sensitive pixels” (ASPs).

An embodiment of the invention is an angle-sensitive pixel devicemanufactured entirely in a standard CMOS fabrication process. In anon-limiting aspect, the ASP device includes a device support structure;a first periodic, light diffracting structure having a period, p₁,disposed in or on a top surface of the support structure; a secondperiodic structure having a period, p₂, oriented parallel to the firstperiodic structure and disposed in the support structure at a selecteddistance below the first periodic, light diffracting structure. As usedherein, m, n are positive integers, λ is the wavelength of amonochromatic, plane, incident wavefront on the first periodic, lightdiffracting structure, and p₁ is equal to or greater than λ. In anaspect, p₁ is equal to p₂. According to an aspect, the second periodicstructure further includes at least two sets of at least two interleaveddiffusion-type diodes, which could be finger diodes. The sets of diodesare, respectively, laterally displaced from the first periodic, lightdiffracting structure by a distance np₁/m, where n can have the values0, 1, 2, 3, 4, 5, 6, 7, 8 and m can have the values 2, 3, 4, 8. The setsof interleaved diodes are disposed at a selected distance,z_(T1)=(m₁/n₁)(2p₁ ²/λ), below the first periodic, light diffractingstructure and the second periodic structure.

An embodied micron-scale device requires both a periodic diffractingstructure to generate Talbot self-images and a structure for analyzingthese self-images. By sizing the entire device to fit within an area ofat most tens of microns on a side, spatial resolution may be achievedthat is comparable with existing image sensors. In an illustrativeaspect, the periodic diffracting structure will have several periodswithin this area to produce an operable periodic self-image. Thus thediffracting structure may have a period of only a few wavelengths.Contemporary planar photolithography techniques can easily achieve theresolution required to generate appropriate diffracting structures.Numerical modeling and simulation can accurately predict behavior forfinite gratings built on a single-micron scale.

According to a general embodiment, the structure for analyzing theself-images generated by the periodic diffracting structure may be anintegrated light detector; for example, at least two, periodic,interleaved sets of at least two diffusion-type diodes as are well knownin the art. According to further embodiments described herein below, thestructure for analyzing the self-images may be one or more layers ofperiodic structures followed by a sensor in the form of at least two,periodic, interleaved sets of diffusion-type diodes, one or more single,large, well-type diodes known in the art, or a combination of theinterleaved diffusion diodes disposed (and partially enclosed) in theone or more single, large, well-type diodes. The one or more layers ofperiodic structures may or may not be arranged co-perpendicularly.

An embodiment of the invention is a lens-less, angle-sensitive pixel(ASP) device that includes a device support structure; a first periodic,light diffracting structure having a period, p₁, disposed in or on a topsurface of the support structure; a second periodic structure having aperiod, p₂, oriented parallel to the first periodic, light diffractingstructure and disposed in the support structure at a selected distancebelow the first periodic, light diffracting structure; and a sensordisposed in the support structure at a selected distance below the firstperiodic, light diffracting structure and the second periodic structure.

An embodiment of the invention is a lens-less light-field detector thatincludes a detector support structure; a first pixel device, and asecond pixel device disposed linearly adjacent the first pixel device.The first pixel device comprises a first periodic, light diffractingstructure having a period, p₁, disposed in or on a top surface of thesupport structure; a second periodic structure having a period, p₂,oriented parallel to the first periodic, light diffracting structure anddisposed in the support structure at a selected distance below the firstperiodic, light diffracting structure, wherein the second periodicstructure is not laterally displaced from the first periodic, lightdiffracting structure; and a first sensor disposed in the supportstructure at a first selected distance below the first periodic, lightdiffracting structure and the second periodic structure. The secondpixel device comprises a first periodic, light diffracting structurehaving a period, p₁, disposed in or on a top surface of the supportstructure; a second periodic structure having a period, p₂, orientedparallel to the first periodic, light diffracting structure and disposedin the support structure at the selected distance below the firstperiodic, light diffracting structure, wherein the second periodicstructure is laterally displaced from the first periodic, lightdiffracting structure by an amount (m₂/n₂)p₁; and a second sensordisposed in the support structure at the first selected distance belowthe first periodic, light diffracting structure, wherein m, n arepositive integers, λ is the wavelength of an monochromatic, plane,incident wavefront on the first periodic, light diffracting structure,p₁ is greater than λ. According to an aspect, the first and second pixeldevices further comprise a first intermediate periodic, lightdiffracting structure having a period, p₁, disposed between the firstperiodic, light diffracting structure and the second periodic structure,oriented perpendicularly to the first and second periodic structures;and a second intermediate periodic, light diffracting structure having aperiod, p₂, disposed between the second periodic structure and the firstand second sensors, oriented perpendicularly to the first and secondperiodic structures, wherein in the first pixel device, the first andsecond intermediate periodic, light diffracting structures are notlaterally displaced from the respective first and second periodicstructure, further wherein in the second pixel device, the first andsecond intermediate periodic, light diffracting structures are laterallydisplaced from the respective first and second periodic structures by anamount (m₂/n₂)p₁. According to an aspect, the detector further comprisesat least an n^(th) (n≧3) pixel device disposed linearly adjacent the(n^(th)−1) pixel device, including a first periodic, light diffractingstructure having a period, p₁, disposed in or on a top surface of thesupport structure; a second periodic structure having a period, p₂,oriented parallel to the first periodic, light diffracting structure anddisposed in the support structure at the selected distance below thefirst periodic, light diffracting structure, wherein the second periodicstructure is laterally displaced from the first periodic, lightdiffracting structure by an amount (m_(n)/n_(n))p₁, where(m_(n)/n_(n))>(m_(n-1)/n_(n-1)); and an n^(th) sensor disposed in thesupport structure at the first selected distance below the firstperiodic, light diffracting structure. In a further aspect, every n^(th)(n≧3) pixel device further comprises a first intermediate periodic,light diffracting structure having a period, p₁, disposed between thefirst periodic structure and the second periodic structure, orientedperpendicularly to the first and second periodic structures; and asecond intermediate periodic, light diffracting structure having aperiod, p₂, disposed between the second periodic structure and then^(th) sensors, oriented perpendicularly to the first and secondperiodic structures, wherein in every n^(th) (n≧3) pixel device, thefirst and second intermediate periodic, light diffracting structures arelaterally displaced from the first periodic structure by an amount(m_(n)/n_(n))p₁, where (m_(n)/n_(n))>(m_(n-1)/n_(n-1)).

Another embodiment of the invention is a lens-less light field imagingdevice comprising a two-dimensional, M×N array of ASP-light-fielddetectors as set forth herein, where M, N are integers equal to orgreater than one.

According to all of the foregoing embodiments, the periodic diffractingstructures may be of various forms including, but not limited to,diffraction gratings, parallel wire arrays, Ronchi rulings, phasegratings, and others well known in the art. Diffracting apertures may bein the form of slits or other aperture shapes, or mismatched refractiveindices. Gratings may advantageously be made of metal or, in the case ofphase gratings, CMOS process-compatible materials (e.g., silicondioxide). The sensor(s) may be, without limitation, reverse-bias p-njunction diodes, forward-biased diodes, p-i-n diodes, charge-coupleddevices (CCDs), single-photon avalanche diodes, or pairs of interleavedN+/p-substrate diffusion diodes. A device may incorporate one or morecolor filters if, for example, the incident light has a broad spectrumthat may advantageously be narrowed.

An embodiment of the invention is an angle-sensitive pixel (ASP) device,comprising a substrate, a phase grating having a period, p₁, disposed inthe substrate, wherein the phase grating is characterized by a periodicvariation of refractive index in a direction transverse to an incidentlight field and forms a periodic intensity pattern in a Talbot planebelow the phase grating, an amplitude transmission analyzer gratingdisposed in the Talbot plane, and a sensor disposed at a selecteddistance below the analyzer grating. In various non-limiting, exemplaryaspects:

the ASP device further comprises a dispersive material patterned overthe phase grating;

the phase grating comprises silicon dioxide;

the sensor comprises at least one pair of interleaved N+/p-substratediffusion diodes.

An embodiment of the invention is an angle-sensitive pixel (ASP) device,comprising a substrate, a phase grating having a period, p₁, disposed inthe substrate, wherein the phase grating is characterized by a periodicvariation of refractive index in a direction transverse to an incidentlight field and forms a periodic intensity pattern in a Talbot planebelow the phase grating, and a sensor disposed at a selected distancebelow the analyzer grating. In various non-limiting, exemplary aspects:

the ASP device further comprises a dispersive material patterned overthe phase grating;

the phase grating comprises silicon dioxide;

the sensor comprises at least one pair of interleaved N+/p-substratediffusion diodes.

Another embodiment of the invention is a light field imaging devicecomprising a two-dimensional, M×N array of ASP devices each including aphase grating with and without an analyzer grating, as outlined above,where M, N are integers equal to or greater than one, and a sensor.

An embodiment of the invention is directed to a method for determining adirection of incident light from an object, comprising creating aperiodic, interference pattern of the incident light from the object,detecting the interference pattern and, determining a phase shift of thepattern relative to a reference position based upon the relativeillumination of different diodes.

Embodiments of the invention thus pertain to imaging devices and methodsthat can enable extraction of information relating to thethree-dimensional structure of the object light. Each ASP in the type ofimager described herein may extract the incident angle of light as wellas its brightness. Individual ASPs may be used to localize one or morelight sources (such as for sun tracking, for example). When many suchASPs are combined in an array, such information may be used toreconstruct three-dimensional surfaces, or multiple distinct points in3-D space, which may have application in, e.g., biological imaging. Animaging device according to embodiments of the invention mayadvantageously be built in a standard semiconductor manufacturingprocess such as those used to build microprocessors and present daydigital camera imagers; for example, standard CMOS fabricationprocesses.

It is to be understood that both the foregoing general description andthe following detailed description are merely exemplary of theinvention, and are intended to provide an overview or framework forunderstanding the nature and character of the invention as it isclaimed. The accompanying drawings are included to provide a furtherunderstanding of the invention, and are incorporated in and constitute apart of this specification. The drawings illustrate various embodimentsof the invention, and together with the description serve to explain theprinciples and operation of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a shows a perspective illustration of a light-field imager andhow light from a source strikes each pixel of an array with a distinctincident angle; FIG. 1 b illustrates that if each pixel in an array candetermine the incident angle as well as the intensity of the light itdetects, then array is able to localize a light source in threedimensions, according to an illustrative embodiment of the invention;

FIG. 2 a is a cross sectional, schematic view of a diffraction gratingwith a definition of scale and dimensions; FIG. 2 b shows FDTDsimulations of the Talbot effect for light normally incident on thegrating, and the self images at multiples of the ½ Talbot depth; FIG. 1c is a plot based on an FDTD simulation showing the lateral shift of theself image at the ½ Talbot depth with shifting incident angle from θ=0°to 5°, according to an illustrative embodiment of the invention;

FIG. 3: FDTD simulations illustrating the effect of including ananalyzer grating at the ½ Talbot depth: a) When the peaks of theself-image align with the bars of the analyzer grating, little lightpasses through to a light detector below; b) When the incident angle isshifted so that the peaks align with gaps in the analyzer grating, muchmore light passes to the detector; c) Intensity of detected lightchanges periodically with swept incident angle, according to anillustrative embodiment of the invention;

FIGS. 4 a, b diagrammatically show structures for extracting informationabout diffraction pattern phase according to alternative, exemplaryaspects of the invention;

FIG. 5 a illustrates an ASP device having multiple, adjacent, singledeep-well photodiodes and stacked, offset gratings disposed above (blackdotted lines illustrate relative alignment of the gratings; FIG. 5 bshows simulation results for various offsets: note that the incidentangles that generate peak responses shift proportionally with the offsetof the grating, according to an embodiment of the invention;

FIGS. 6 a, b are microphotographs of a) one ASP, and b) an 8×8 array ofASPs, manufactured in 130 nm CMOS, according to an embodiment of theinvention;

FIG. 7 is a diagrammatic cross sectional view of an image sensoraccording to an alternative aspect of the invention;

FIG. 8 is a perspective view of an ASP-based light field image deviceaccording to an embodiment of the invention;

FIG. 9 is a graph showing measured responses of an ASP as incident angleis swept, according to an illustrative aspect of the invention;

FIG. 10 is a graph showing the measured effect of wavelength on angularsensitivity, b, and modulation depth, m, according to an illustrativeaspect of the invention;

FIG. 11 shows the measured ASP array response to a light source held 500μm above the array and slightly to the left: a) Responses of individualsensors, where brighter squares represent more heavily illuminatedsensors and white lines delimit individual ASPs; b) Computed incidentangle for each ASP (projected into the x-y plane), according to anillustrative aspect of the invention;

FIG. 12 shows how an 8×8 ASP array accurately resolves light sourcelocations in 3-D space: a) The measured light-vector field due to asource 550 μm above the array can clearly reconstruct lateral shifts inlocation (in this case by 100 μm); b) The measured light-vector fieldcan also be used to reconstruct changes in depth (z) of a light source,in this case by 50 μm, according to an illustrative aspect of theinvention;

FIG. 13 is a diagrammatic cross sectional view of an image sensoraccording to an alternative aspect of the invention;

FIG. 14 is a diagrammatic cross sectional view of an image sensoraccording to an alternative aspect of the invention;

FIG. 15 is a diagrammatic cross sectional view of an image sensoraccording to an alternative aspect of the invention;

FIGS. 16 a, 16 b are, respectively, a top view and a cross sectional ofan alternate imaging sensor according to an illustrative aspect of theinvention;

FIGS. 17( a-c) show top cross sectional plan views of a full interleaveddiode light-field sensor cell according to an exemplary embodiment ofthe invention;

FIG. 18 is a diagrammatic cross sectional view in which all of thediodes are shifted by ⅛ of the metal grating pitch, according to anillustrative aspect of the invention;

FIG. 19 graphically shows simulated photocurrents from four distinctdiode arrays at 0, 90 180 and 270 degrees relative to a grating wherethe incident angle was swept from −30 to 30 degrees, according to anillustrative aspect of the invention;

FIGS. 20( a-c) schematically illustrate the use of incident angle datato compute 3-D locations in conjunction with a lens system, according toan exemplary embodiment of the invention;

FIG. 21 is a diagrammatic cross sectional view of an image sensoraccording to an alternative aspect of the invention;

FIG. 22 is a diagrammatic cross sectional view of an image sensoraccording to an alternative aspect of the invention;

FIGS. 23( a-d) illustrate a phase grating based ASP and simulated Talbotimages according to an exemplary embodiment of the invention;

FIG. 24: a) a phase grating based ASP with an analyzer grating; b)simulated Talbot images at normal incidence; c) simulated Talbot imagesat 10 degrees; d) measured response of the detector as a function ofincident angle, according to an exemplary embodiment of the invention;

FIG. 25: a) cross sectional view of a phase grating based ASP withinterleaved finger diodes; b) measured response of the detector as afunction of incident angle, according to an exemplary embodiment of theinvention;

FIG. 26: a) top view and cross section of a pair of interleaved diodes.(b) an ASP that uses interleaved diodes instead of an analyzer gratingand an n-well diode to detect complementary phases of a singlediffraction pattern, according to an illustrative embodiment of theinvention;

FIG. 27: manufacturing of phase gratings in standard CMOS without addedmasks: a) cross section of chip as manufactured, in which top two metallayers (black) have been used as etchstop layers. Both directinterleaved diode ASPs (bottom) and analyzer grating ASPs (top) can bemade; b) applying an anisotropic oxide etch removes oxide insulator(light blue) until metal is exposed; c) removal of the metal etchstopproduces the desired phase grating structures above the photosensors; d)coating the surface with a material with a distinct refractive index(pink) provides an additional parameter in grating design, according toan illustrative embodiment of the invention;

FIG. 28: ASP using interleaved “fingered diodes” directly in place ofanalyzer grating: a,b) top views of grating, diodes, colors indicatedifferent offsets; c) cross section; d) illustration of how shallowlypenetrating photons (green arrows) generate photocurrent in desireddiodes (red) but electrons from deeply penetrating photons can diffuseto either set of fingered diodes (red or blue) blurring anglesensitivity; e) embedding interleaved diodes in a well causes deeperphotons to be absorbed in or near the well's depletion regions (violet)preventing blurring while still detecting all photons for intensitymeasures; f) alternate interleaved diode design with three wider diodesto capture three angles, according to an illustrative embodiment of theinvention;

FIG. 29: Quarter-order Talbot phase gratings: a) pitch of thediffractive layer is twice that of the analyzer layer; b) simulation oflight interacting with device; c) angle sensitivity of quarter-orderTalbot device, according to an illustrative embodiment of the invention;

FIG. 30: Graph showing the effect of a dispersive medium in the phasegrating;

FIGS. 31( a-d): Measured output of four ASP structures showing all ASPstructures achieve a characteristic angle sensitive output response,according to a illustrative aspect of the invention;

FIG. 32: Graph showing the measured variation of modulation depth andangular sensitivity with grating pitch. The highest modulation depth ofm=0.32 is obtained when grating pitch, d=0.86 μm; and

FIG. 33: Chart showing the measured quantum efficiency of four ASPvariants, relative to a conventional active CMOS pixel with ann-well/p-substrate photodiode.

DETAILED DESCRIPTION OF NON-LIMITING, EXEMPLARY EMBODIMENTS OF THEINVENTION

Reference will now be made in detail to the present exemplaryembodiments of the invention, examples of which are illustrated in theaccompanying drawings.

Embodiments of the invention are directed to angle-sensitive pixeldevices and light field image detectors incorporating these ASP devicesto measure both the intensity and the incident angle of incident light,and associated methods. The disclosed apparatus and methods utilize theTalbot effect.

The Talbot effect, or the self-imaging property of periodic objects suchas diffraction gratings, was first discovered by Henry Fox Talbot in1836. When an infinite diffraction grating is illuminated by a planewave normal to its surface, identical images of the grating are formedat certain equally spaced distances behind the grating. FIG. 2 adiagrammatically illustrates the parameters of a diffraction grating 102with incident light 100 (nominally 500 nm wavelength) striking thegrating at normal incidence and at a θ=5 degree incident angle. TheTalbot effect is a consequence of Fresnel diffraction, and theinterference image (diffraction pattern) 104, indicated at arrow 1 asshown in FIG. 2 b, forms at integer multiples of the Talbot distancez_(T)=2d²/λ, where d is the period of the grating and λ is thewavelength of incident light. Additional, more complex sub-images 105,106 can be observed at the fractional Talbot distances z=(m/n)z_(T)(indicated at arrows 1/2, 3/2), where m and n are positive integers.

A property of the Talbot effect is its response to off-axisillumination. For macroscopic (d>>λ) linear gratings illuminated by anoff-axis plane wave incident at angle θ, self-imaging is observed atmultiples of the distance z=2 cos³(θ)d²/λ. Furthermore, the imagesexhibit a lateral shift Δx=z tan(θ) perpendicular to the grating linesas a result of the off-axis wave propagation.

Multiple sources of off-axis illumination each generate their own set oflaterally shifted grating self-images, and these self-images superpose.For small angles, these self-images all form at approximately the samedistances, and the superimposed image contains information about themagnitude of illumination as well as direction. The incident angles oflight rays striking the grating can be determined by measuring the shiftin Talbot self-images.

FIG. 2 c graphically shows the light intensity of the Talbot images at adiode plane for normally incident light and light incident at θ=5degrees. The lateral shift of the diffraction patterns changes withincident angle.

Modern semiconductor manufacturing of standard CMOS circuits allows forthe construction of very fine features, on the order of a singlewavelength of light, and so allows for the construction of metaldiffraction gratings and photodiode arrays that are smaller than thewavelength of visible light. To generate the Talbot effect in a standardCMOS layer stack, the self-images need to form within microns of thediffraction grating. This may require the diffraction grating to have aperiod of only a few wavelengths. Conventional analyses of diffractionare invalid at these dimensions; however, numerical simulations such asthose used to generate FIG. 2, confirm that, even for these geometries,diffraction generates Talbot-like self-images at regular distances.These periodic intensity patterns retain incident angle sensitivity.

The challenge, then, is to extract shifts in these periodic intensitypatterns using structures on a pixel scale. For macroscale applications,the simplest approach to measuring these shifts is to place a smallarray of CCD or CMOS photosensors at the plane of self-image formation.The array captures the self-image directly, which can be used todetermine the angle and intensity of incident light. At the microscale,however, the penetration depth of light in silicon limits the resolutionof photodiodes to about 1 μm, making it difficult to resolve sub-micronfeatures of the self-image.

A micron-scale light-field imager device requires both a Talbotself-image generator and a structure that can analyze these images. Inorder to achieve spatial resolution comparable with existing imagesensors, the entire device structure must fit within an area that is atmost tens of microns on a side. To produce a reasonably periodicself-image, the grating must have several periods within this area.Together these two constraints suggest using a grating with a period ofonly a few wavelengths. Contemporary planar photolithography techniquescan easily achieve the resolution required to generate appropriatediffraction gratings. Numerical modeling and simulation can accuratelypredict behavior for finite gratings built on a single-micron scale.

Numerical treatments show that as long as the period is greater than thewavelength of incident light, Talbot-like self-images can be observed inclose proximity to a diffraction grating. We have performed simulationsusing the finite-difference time domain (FDTD) technique and observedpatterns as shown in FIGS. 2 b and 2 c. In particular, starting from the½ Talbot distance, we observe strong intensity patterns with periodicityidentical to the diffraction grating. Additional simulations show thatunder off-axis illumination, the intensity patterns generated by thehigh-density gratings shift laterally. An effect of moving towavelength-scale diffraction gratings is to suppress higher-orderfractional Talbot images.

To extract incident angle information about the Talbot pattern, it isnecessary to characterize the horizontal offset of the self-images.Previously reported work used gratings (and self images) that weresignificantly larger (pitch of d=250 μm) than the pixels of the imagesensor itself. Thus the image sensor array could directly capture theself-image as a set of electrical signals. However, in a micron-sizedevice according to an aspect of the invention, the high density imagerarray would require a pixel pitch of ¼ the grating pitch (e.g., on theorder of 200 nm) to effectively resolve the features of the Talbotimage. Although sub-micron photosensors can be built, the images theycapture tend to be blurred by diffusion effects, limiting their actualresolution to 1 μm or worse.

A solution provided by an embodiment of the invention includes a secondparallel analyzer grating 304 of identical period to the first grating302 disposed at the self-image plane, as illustrated in FIGS. 3 a, 3 b.The second (analyzer) grating 304 uses the Moiré effect to filter theTalbot image. When the intensity peaks align with gaps in the secondgrating as shown in FIG. 3 b, light passes through the analyzer grating304. When the intensity peaks are out of alignment (FIG. 3 a), the barsof the analyzer grating block the light. By placing a single largephotosensor under the analyzer grating and measuring the total lightflux, we can extract the alignment of the self-image with the analyzergrating (FIG. 3 c).

FIG. 4 b shows an exemplary, diagrammatic illustration of such astructure embodiment 300-2 for extracting partial information about thediffraction pattern phase. Two metal gratings 302 a, 302 b are placed ata 90 degree lateral offset relative each other over a single, largewell-diode 307 integrated into substrate 310. Separate pixels withgratings shifted by 0, 180 and 270 degrees or, alternatively, 0, 120 and240 degrees, for example, would extract full angle information. Thisapproach decouples the design of the diodes from that of the gratings,allowing for better diodes. Also, because the finest features in thisaspect are the gratings themselves rather than the photodiodes, the sameclass of structure can be built using lower resolution photolithography(i.e., in a larger feature size, cheaper manufacturing process).

The total light flux detected is dependent on both the overall sourcebrightness and the incident angle. This may lead to an ambiguity betweenintensity and angle in the sensor output, since a bright source at ablocked angle yields the same sensor output as a dimmer source at anangle passed by the analyzer grating. To disambiguate angle andintensity, in accordance with an aspect of the invention as illustratedin FIG. 5 a, a detector 400-2 includes n (n=4 as shown) singlewell-diode sensors 407 _(n) integrated in substrate 410, and two stackedgratings 402 a, 402 b disposed above in close proximity so that they seeapproximately the same light field. Each diode has a different relativeoffset between the analyzer grating 402 b and the image-generatinggrating 402 a. Using the unique signals produced by each of the set ofsensors, one can recover intensity and incident angle.

Simulated responses for one set of four sensors under plane illuminationof different angles are shown in FIG. 5 b. It is seen that thetransmission through the analyzer grating is periodic in incident angledue to the lateral shift of the periodic self-images. The responses ofthese sensors can be approximately modeled by the equations:

R ₀ =I ₀(1−m cos(bθ))F(θ)

R _(1/4) =I ₀(1+m sin(bθ))F(θ)

R _(1/2) =I ₀(1+m cos(bθ))F(θ)

R _(3/4) =I ₀(1−m sin(bθ))F(θ)  (1)

where I₀ is proportional to incident intensity, θ is incident angle, mis a measure of the modulation depth, and b is a measure of angularsensitivity. F(θ) is an even-symmetric function included to account forsurface reflections and other effects that reduce responses to highangle incident light independent of angular sensitivity.

From the four outputs in equation 1, it is possible to determine theintensity and incident angle (in the x-z plane) of light. Summing theASP responses R₀ and R_(1/2) (or R_(1/4) and R_(3/4)) removes themodulation produced by incident angle and provides information onoverall intensity.

$\begin{matrix}{{I_{0}{F(\theta)}} = {\frac{R_{0} + R_{\frac{1}{2}}}{2} = \frac{R_{\frac{1}{4}} + R_{\frac{3}{4}}}{2}}} & (2)\end{matrix}$

Incident angle can be extracted as:

$\begin{matrix}{\theta = {\frac{1}{b}{\tan^{- 1}\left( \frac{R_{\frac{1}{4}} + R_{\frac{3}{4}}}{R_{\frac{1}{2}} - R_{0}} \right)}}} & (3)\end{matrix}$

Because the lateral shift of the Talbot images is observed only foroff-axis illumination at angles perpendicular to the grating lines, thedevice 400-2 is responsive only to angles in one direction. In order toobtain full illumination angle information, a second set of identicaldevices with gratings rotated by 90 degrees, in close proximity to thefirst, were provided. This second set is responsible for measuring theangle information ignored by the first set of sensors. A completeangle-sensitive pixel (ASP) 500-1 composed of eight different sensorsplaced in close proximity is shown in FIG. 6 b. Four sensors areresponsible for the angle in the x-z plane; four more are needed for theangle in the y-z plane. For both x-z and y-z gratings,diffraction-analyzer offsets of 0, d/4, d/2 and 3d/4 were used. Theanalyzer gratings were positioned at the ½ Talbot distance, the smallestdistance where self-images with periodicity identical to the diffractiongrating are found. An 8×8 ASP array light field image sensor 500-2manufactured in a digital 130 nm CMOS fabrication process is illustratedin the photomicrograph of FIG. 6 a.

The overall size of the exemplary eight-sensor ASP 500-1 is 20 μm by 40μm, with each individual sensor being 10 μm square. The stackeddiffraction gratings were built in wiring layers, above intrinsic p-njunction photodiodes. In this illustrative example, each grating in eachof the eight sensors was a Ronchi ruling (equal width bars and gaps)using copper bars with a period of 880 nm. All other space was filledwith silicon dioxide. One set of gratings was used to bus out the datagenerated, which eliminated the need for wiring lanes in the array. Asthe gratings provide a large number of bus lines, the eight ASP outputsare read in parallel. The grating separation, z, was limited byavailable interconnect layer spacing, and pitch d chosen from numericalsimulations to maximize modulation depth, m, for green (525 nm invacuum, 350 nm in oxide) light. For the device 400-1 shown in FIG. 5 a,empirical simulations for green (λ=525 nm in vacuum) light determinedthe ½ Talbot distance in silicon dioxide to be 2 μm. The top diffractiongrating was positioned in the 6th metal layer and the analyzer gratingin the 3rd metal layer, for a separation of 2 microns. A single p-nphotodiode in each of the eight sensors measured the total light fluxthrough the stacked gratings. A standard 3T active pixel sensor was usedto buffer the photodiode outputs, and several multiplexers allowedaccess to each ASP individually.

FIG. 7 shows an illustrative aspect of a device embodiment 6000-1similar to that shown in FIG. 5 a in which three single diodes 6007,6008, 6009 are disposed adjacent two grating layers 6001 a, 6001 b.Second grating layer 6001 b is shifted relative to grating 6001 a by 0,⅓ and ⅔ of the grating period.

According to an alternative aspect, a light field image detector 600-1is illustrated in FIG. 8. In this aspect, a second set of gratings 502a, 502 b rotated by 90 degrees and interleaved between grating 402 a,402 b are provided in close proximity thereto. This second set ofgratings is responsible for measuring the angle information ignored bythe first set of sensors.

To test our ASP, a light source (commercial green LED, with centerwavelength of 525 nm and spectral width of 32 nm) was mounted on avariable angle arm at a fixed distance from the fabricated arrays. Noadditional collimation or filtering was performed, as a non-idealillumination source better approximates real-world imaging applications.When a range of wavelengths are present, the self-images observed are asuperposition of the intensity patterns produced by each wavelength. Thespectral width of the source is relatively narrow and the path lengthdifferences, which make the Talbot patterns, are shorter than thesource's coherence length, so we did not expect significant deviation inperformance from our monochromatic, coherent simulations.

We recorded the outputs of a single ASP for each angle as the source wasmoved. The outputs corresponding to one set of four sensors in the ASPare shown in FIG. 9. Reasonable agreement was obtained between measuredresults and those predicted by simulation. Fitting the curves in FIG. 9with the model in equation (1) gives b=15 and m=0.7, with aroot-mean-squared error of 9%. The second set of four sensors (forcharacterizing angles in the y-z plane) produced similar curves inresponse to changes in incident angle. Differences observed betweenmeasurement and idealized simulations such as those in FIGS. 2 and 3 aredue to reflection off the silicon dioxide surface, manufacturingvariation, and the finite gratings actually used. However, oursimulations reasonably characterized the angular sensitivity andmodulation depth of the ASP.

Fine-pitch gratings are known to polarize the light they transmit. Arecent study on the polarization-dependent Talbot effect in high-densitygratings predicts that gratings with period of approximately 2.5λ shouldshow significant polarization sensitivity. Specifically, the Talbotself-images formed at the ½ Talbot distance by TE (electric fieldparallel to the grating lines) polarized light should be approximatelytwice as bright as those formed by TM (magnetic field parallel to thegrating lines) polarized light. Our observations are in good agreementwith this prediction: when we rotated the polarization of the incidentlight on our ASP from TE to TM, the overall observed intensity decreasedby a factor of 2.05. However, both angular sensitivity b and modulationdepth m changed by less than 10%. These characteristics indicate thatthe TM-polarized Talbot self-images are weaker than the TE-polarizedself-images, but otherwise behave similarly in their encoding of angleand intensity information.

The design was optimized for λ=525 nm, but we tested it across a rangeof wavelengths from 400 nm to 620 nm. We expected little change in anglesensitivity b in response to changes in wavelength, as the Talbotself-images do not change in periodicity with changes in λ. Thisprediction was born out by measurement, as can be seen in FIG. 10: b wasonly weakly sensitive to λ over the range 400 nm to 620 nm. However,changes in wavelength significantly change the Talbot distances. Theanalyzer grating was not optimally positioned when λ≠525 nm, so theobserved self-images were blurred, and modulation depth, m, degraded.Over this range of wavelengths, we recover angle information lessefficiently, but the angle sensitive function does not vanish. The factthat the ASP works across such a range of wavelengths is a directconsequence of analyzing the self-image at the ½ Talbot distance, wherethe relative depth of the Talbot pattern is least sensitive to λ.

To confirm the light-field imaging capability of our sensors, we placeda multimode fiber tip 500 μm directly above the ASP array. Aftercoupling light from a light emitting diode (identical to the one used insingle ASP tests) into the fiber, light exiting the fiber will have aconical profile, and thus a simple divergent light field at the plane ofthe array. We recorded from all 64 sites on the ASP array and measuredthe output of each sensor, as shown in FIG. 11 a. As can be seen,adjacent sensors tuned to different angles responded very differently,and their relative responses depend upon their overall location relativeto the light source. Applying equation (3) and the angle response datashown in FIG. 9, we reconstructed the light vectors for each ASP, asshown in FIG. 11 b.

To further confirm the capabilities of our array, we moved the lightsource to various locations in three-dimensional space above the array.At each position we recorded the sensors' responses and reconstructedthe incident angle of light coming from the fiber. The array could beused to accurately reconstruct the location of the light source in twodimensions, as shown in FIG. 12 a, where the source was moved by 100 μmin the x-direction, and the computed incident angles reflect this. Morestrikingly, the array could be used to accurately localize the lightsource in the third, z direction, accurately capturing a 50 μm shift inthe height of the source above the array, as shown in FIG. 12 b. Thus anarray of ASPs is able to accurately reconstruct the three-dimensionalstructure of simple light sources, providing information beyond what isavailable from the intensity map available from a standard image sensor.

FIG. 4 a shows a cross sectional schematic of a non-limiting exemplarydevice embodiment 300-1 of the invention. The device includes a metalslit grating 301 and a substrate 310 with multiple linear arrays of twointegrated, interleaved fingered diodes (A) 307, (B) 308 that arerelatively shifted by 180 degrees (i.e., offset by zero and one-halfperiod of the grating) relative to the grating. Multi-finger diodesadvantageously provide maximum photon capture.

FIG. 13 shows an imaging device 500-1 based upon a three-diode structureaccording to a non-limiting exemplary embodiment of the invention. Thedevice includes a metal transmission grating 501 having multiple,periodic slit apertures 503. The light shadings indicated by referencenumeral 504 do not represent any physical part of the device, rathermerely the alignment between the grating and the diodes. The devicefurther includes a single structure of three linear arrays of threeinterleaved diodes, 507, 508, 509, integrated in a substrate 510. Thethree illustrated diodes of diode array 507 are aligned with the grating(no offset) and thus will detect a zero degree phase shift in theinterference pattern (not shown). Similarly, the three illustrateddiodes of diode array 508 are offset by ⅓ of the grating period and thusdetect a 120 degree phase shift; while the three illustrated diodes ofdiode array 509 are offset by ⅔ of the grating period and thus detect a240 degree phase shift.

FIG. 14 shows an alternate device arrangement 400-1 of diodes andgrating according to a non-limiting, exemplary embodiment. As shown inFIG. 14, an integrated, single-interleaved set of four diodes 407, 408,409, 411 are positioned offset by zero, ¼, ½ and ¾ of the grating 401period providing respective phase shifts of 0°, 90°, 180° and 270°. Notethat the single-interleaved set of four diodes is different than, e.g.,the two adjacent diode sets as shown in FIG. 4 a. Again, the lightshadow areas in the figure do not reflect any additional physicalstructure; rather, they only indicate alignment between diodes andmetal.

FIG. 15 shows a cross-sectional schematic of a detector device 100-1according to an alternative, non-limiting exemplary embodiment of theinvention. Sensor device 100-1 incorporates one set of interleaveddiodes 121 at 0 and 180 degrees relative to the grating and another setof interleaved diodes 123 at 90 and 270 degrees. This type ofarrangement may prevent diode overlap. The two shifted gratings 101 a,101 b are shown as darkly shaded while the lighter shading 103 beneatheach grating is presented merely to show the alignment between thegrating and the diodes and does not represent any physical structure ofthe device.

FIG. 16 a shows a top view of components of a non-limiting, exemplarydevice aspect 8000-1 incorporating just two slits 8003 and two diodes8007, 8008. FIG. 16 b is a cross-sectional view through the dashed linein FIG. 12 a. This design is compact, allowing for higher spatialresolution.

FIGS. 17( a-c) show top views of components of a non-limiting, exemplaryfull interleaved diode light-field sensor cell 5000-1, having gratingsand diodes in both the vertical (5006 a, b) and horizontal (5006 c, d)orientations, which may be used to capture both azimuth and altitudeinformation about the source object. FIG. 17 a illustrates the layout ofdiodes (e.g., n-type diffusion in p-substrate), wherein each set ofvertically-oriented diodes 5006 a, b contains a pair 5002, 5004 ofinterleaved diode arrays and each set of horizontally-oriented diodes5006 c, d contains a pair 5008, 5010 of interleaved diode arrays. FIG.17 b shows the associated metal gratings 5001 with the same orientationsas the corresponding diode arrays. As further shown in FIG. 17 b, thevertically-oriented gratings may be used as a data bus to carryinformation from each column to the edge of the array at 5015 withoutgiving up area in the imager itself. Alternatively, the gratings may beused to tie many individual diodes to processing circuits away from thearray itself. This maximizes the photosensitive area of the imager,recovering area lost to circuits required to convert light intoelectrical signals. FIG. 17 c shows an overlay of diodes and gratingsshown in FIGS. 17 a, b. FIG. 15, which shows a cross-section of thehorizontally-oriented detectors (i.e., along the dashed black line inFIGS. 17 a-c, illustrates that the relative alignment of the diodes andmetal grating are shifted for the 0/180° cases versus the 90/270° case.

FIG. 18 shows an exemplary device aspect 2000-1 similar to that of 100-1except that all diodes are shifted by ⅛ of the grating pitch,representing phase shifts of −45°, 135°, 45° and −135°. This figureillustrates that the precise alignment of diodes and metal is lessimportant than the relative alignment of diodes to each other. The ⅛period shift should have no appreciable effect on the function of thestructures disclosed herein. This insensitivity applies to allstructures disclosed herein, and to the alignment of secondary gratingsin the “double grating” embodiments described herein.

FIG. 19 graphically shows simulated photocurrents from four distinctdiode arrays at 0, 90 180 and 270 degrees relative to a grating wherethe incident angle was swept from −30 to 30 degrees, according to anillustrative aspect of the invention. As can also be seen from FIG. 19,each diode shows multiple peaks, indicating that equations (1) may notnecessarily lead to a unique angle extraction. This may be remedied byusing multiple structures with different grating geometries (andtherefore different values of “k”), placed adjacent to each other. Ifthe mapping from incident angle to diode response is different, thendifferent peaks of activity may be distinguished. This may thenfacilitate construction of a sensor that is able to cover the entirerange of incident angles.

In the ideal case where each diode is responding to exactly the sameincident angle of light, one may expect some redundancy in the responsesin the eight diode aspect described above. For example,

D0+D180=D90+D270,

implying that maintaining all four separate signals may be redundant.This redundant information may be removed by redefining the response interms of three numbers:

D0−D180,

D90−D270,

and

D0+D180+D90+D270.

This recoding could be performed in either the analog or digital domainon the same integrated circuit as the sensors and gratings.

If incident angle is not constant across the imager (as would be thecase in FIGS. 1 b and 20 a-c), then adjacent gratings will not seeidentical incident angles. Since the four diodes, D0, D90, D180 and D270are not all interleaved with each other, but appear in adjacent pixels,they may encode slightly different incident angles, and so contain somenon-redundant information that would be lost in recoding. Nonetheless,recoding signals can provide benefits by allowing for differentweighting of different components of the data before conversion todigital signals or before transmission off chip.

FIG. 21 shows a device embodiment 700-1 similar to that of device 100-1in FIG. 15, with the exception that the two sets of interleaveddiffusion-type diodes 121, 123 (721, 723) are, respectively, disposed intwo single, large well-diodes 724, 725. According to this aspect,crosstalk observed in the sub-micron size diffusion-type diodes may bereduced, since the large photodiodes collect the electrons and holesgenerated by photons that penetrate into the substrate beyond the thin,interleaved diodes. Thus the large well diodes are fabricated deepenough to enclose the interleaved diodes but shallow enough to catchelectrons.

Interleaved/deep-well diodes can also be incorporated into devices whichrely on multiple sets of gratings for angle sensitivity. An exampledevice 800-1 using two gratings placed ½ Talbot distance apartvertically and photodiodes at the 1^(st) Talbot distance is shown inFIG. 22. As described above, the large-well photodiodes measure thetotal light flux passing through the two gratings. In this aspect, themode of operation is identical to that of the basic multiple gratingdevice. However, when the incident angle is such that the light fluxthrough the grating stack is strong, the interleaved photodiodes helplocate the lateral offset of the periodic intensity pattern with greateraccuracy. This provides improved characterization of incident anglearound a set of known angles without interfering with basic function.

As detailed above, Talbot self-images are used to perform incident anglemeasurements. Modern semiconductor manufacturing is used to form amicron-scale, fine-pitch transmission amplitude grating that createsinterference patterns (Talbot images) from light impinging upon it.Changing the light field incident angle causes these interferencepatterns to shift laterally, which can be detected by a detector.Characterizing these shifts allows one to determine incident angle at apixel scale. However, the above disclosed apparatus and method ofincident angle measurement using a metalized top grating significantlyreduces sensitivity of the detector to local light intensity. The metalgratings used to generate the interference patterns block a significantfraction of the available light. While reduced light sensitivity is nota significant problem for many applications, maintaining highsensitivity comparable to that of a traditional photodetector permitsmore widespread deployment of angle-sensitive imagers.

According to an embodiment that can mitigate this loss of sensitivity, amicron-scale phase grating formed at a pixel scale in an image sensorchip is used in place of the top, amplitude transmission grating. Aphase grating similarly creates an interference pattern and thereforegenerates Talbot self-images, which we characterize in a manner similarto the embodiments disclosed above. In using phase gratings, we replacethe top metal (e.g., wire grid) diffraction grating with athree-dimensional structure (i.e., phase grating) containing materialspossessing a different refractive index from that of the sensor chipsubstrate. This provides significantly increased light sensitivity asthere are no wires to block light striking the pixel.

A simple phase grating 2300 can be formed using crenellated structuressuch as those shown in FIG. 23 a. The shaded area 2310 is a material ofrefractive index n₁, while the white area 2312 is a material ofrefractive index n₂. Incident wavefronts (arrow) which pass through thepeaks of the crenellations exhibit a phase shift relative to those whichpass through the valleys. By Huygens' Principle, the phase shiftdifferences and the path length differences predict periodic intensitypatterns similar to those observed with transmission amplitude gratings(FIG. 23 b). The location of these patterns is influenced by theillumination wavelength and (phase) grating pitch. As shown in FIGS. 23c, d, the intensity patterns shift laterally as the incident angle ofilluminating light changes.

As disclosed above, information about the angle of incident light can beextracted from the lateral shifts in the diffraction pattern. However,the analyzer grating of a single ASP samples only one phase of theperiodic intensity pattern; thus the output of a single angle-sensitivepixel cannot distinguish between changes in intensity and incidentangle. The differential signal from a pair of ASPs whose phases (α's)differ by π is required to unambiguously recover angle information.Exemplary angle sensitive pixel arrays disclosed hereinabove used fourASPs, where each ASP's response has identical m and β parameters butdistinct values for α (α=0, π/2, π, 3π/2) to obtain a full quadraturedescription of incident angle.

However, since several ASPs are required to completely characterizeangle, angle information is captured at a significantly reduced spatialresolution as compared to intensity information. In the above disclosedembodiments, the metal gratings used to achieve angle sensitivity blocka significant fraction of incident light from the photodiode. As aresult, the quantum efficiency (QE) of the above embodied ASP devices is6-10 times less than an equivalent, exposed photodiode without gratings.This reduced sensitivity limits the usefulness of angle-sensitive pixelsin low-light and high speed imaging applications.

FIG. 24 a illustrates one exemplary ASP device 2400 and technique, witha periodic, rectangular binary phase structure 2300 implemented in anintermetal dielectric layer stack, and an amplitude grating 2420positioned below (i.e., optically downstream) the phase grating 2300 inthe plane where strong intensity (Talbot) patterns are created by thephase grating. As the intensity patterns shift due to changing incidentlight angle, they align with either the bars or the gaps of theamplitude grating 2420. When intensity peaks align with the bars asshown in FIG. 24 b, most of the light is blocked and a detector 2430located below the analyzer grating registers a weak response. When peaksalign with the gaps as shown in FIG. 24 c, most of the light passesthrough to the detector below. This produces the alternating strong andweak response shown in FIG. 24 d.

FIG. 25 a illustrates another embodiment of the invention. The ASPdevice 2500 includes the phase grating 2300 as illustrated in FIG. 24 a.In this embodiment, however, there is no analyzer grating; rather, aninterleaved collection (e.g., a pair) of small detectors 2550 arelocated in the plane containing the strong intensity (Talbot) patternsproduced by incident light interacting with the phase grating. Thesedetectors directly measure the patterns as they shift in response toincident angle. Intensity maxima on the detectors result in strongresponse, while intensity minima result in weak response, as illustratedin FIG. 25 b. More specifically, the detectors 2550 are a pair ofinterleaved N+/p-substrate diffusion diodes. The outputs of this diodepair record complementary phases of the Talbot pattern. This structure,therefore, enables the direct capture of angle information whileeliminating the lower metal (analyzer) grating as shown in FIG. 24 a.FIGS. 26 a, b show a schematic structure of an ASP that uses a pair ofinterleaved N+/p-substrate diffusion diodes and a Finite-DifferenceTime-Domain (FDTD) simulation of the capture of the complementary phasesof a single diffraction pattern.

The effects of grating pitch, height, and index mismatch on the depth,strength, and angle sensitivity of resulting Talbot patterns allinfluence the Talbot effect for phase gratings at the scale of only afew wavelengths.

One of the major benefits of the parent invention amplitude-gratingbased ASPs is that they can be made in completely standard CMOS with nopost-processing. This has a variety of benefits, such as low cost andease of manufacture. Unfortunately, a standard CMOS layer stack providesno ready-made components for phase gratings similar to the wires usedfor amplitude gratings.

Techniques from CMOS MEMS (see, e.g., Fedder, Santhanam et al. 1996;Fedder 1997) can be employed to build phase gratings by using metalstructures as etch stops for anisotropic oxide etches. Using theavailable metal layers permits for high-precision phase gratingfabrication without additional masks or patterned depositions. A summaryof the process is illustrated in FIG. 27 for both interleaved diode ASPs(bottom) and analyzer grating ASPs (top). FIG. 27 a illustrates thecross section of a chip as manufactured, in which top two metal layers(black) have been used as etchstop layers. Starting with a standard CMOSstack of patterned metal embedded in silicon dioxide, we then use a deepreactive ion etch (DRIE) to remove silicon dioxide, stopping on themetal layers, as in FIG. 27 b. This oxide etch is followed with a metaletch, which leaves a patterned interface between the oxide and adjacentmaterial (e.g., air, unless subsequent layers are added; FIG. 27 c). Byappropriately designing the metal etch stops that define the tops andbottoms of the phased gratings, we can control the pitch and alignmentof the gratings. Choosing which pairs of metal layers to use providessome control of grating height.

An alternate approach that yields a similar result is to use only onegrating as an etch stop (to define the high parts of the phase-gratingcrenellation), and use a timed oxide DRIE to etch out the low parts ofthe phase grating. Once again, this is followed with a metal etch toexpose the high parts and generate a pure phase grating.

This basic post-processing requires only a few steps and no precisionalignment, so the additional cost beyond that of the manufactured chipwill be low. Additional steps may be advantageous, such as a bulk etchremoving chemically distinct passivation layers. Protecting theinput/output pads of the chip may also require a coarse alignment stepand a low-resolution mask. Nevertheless, using the CMOS stack as asubstrate for phase grating fabrication significantly simplifiesmanufacturing.

Alternatively, other methods of generating phased gratings can beemployed, such as nano-imprint lithography, or direct patterning usingperiodic masks and standard photolithography.

The relative refractive index at the grating interface can be modified(from the 1:1.4 of air and SiO₂) with a conformal coat of additionalmaterial, as illustrated in FIG. 27 d; for example, adding parylenewould provide a 1.66:1.4 interface and provide protection from water.Altering the change in refractive index provides further control overthe precise location of the self-images beyond that available throughgrating pitch and height. Preliminary simulations indicate that theheights (˜1 μm) available in standard CMOS, and available interfaceindices (air, parylene) should provide reasonable performance. Inaddition, deliberately choosing a dispersive material will make itpossible to generate wavelength-independent phase shifts, reducing thesensitivity of the Talbot effect to wavelength.

As disclosed hereinabove, another source of reduced sensitivity is theanalyzer grating. Regardless of the (top) grating used to generate theTalbot self-images (amplitude or phase grating), the (bottom) analyzergrating will still block light that is out of alignment. However, if thelateral shift of the Talbot self-images is directly detected byphotodiodes, the analyzer grating is no longer necessary.

One simple approach for accomplishing this is illustrated in FIGS. 28a-c. By interleaving two distinct sets of photodiodes, one can detectwhether the self-image is in phase or precisely out-of phase with theprimary grating. Using a second set of such interleaved photodiodes at a¼ pitch offset results in a set of outputs analogous to those shown inFIG. 5 for stacked amplitude gratings. Ideally this would provide betterquantum efficiency while requiring half as much area for a given ASP,since two measurements would result from each structure.

This approach requires very fine resolution photodiodes on the scale of0.5 μm pitch. FIG. 28 illustrates a phase grating-based ASP usinginterleaved “fingered diodes” in place of an analyzer grating. FIGS. 28a, b are top views of the grating and diodes, where the different colorsindicate different offsets. FIG. 28 c shows the ASP device in crosssection. FIG. 28 d illustrates how shallowly penetrating photons (greenarrows) generate photocurrent in desired diodes (red) but electrons fromdeeply penetrating photons can diffuse to either set of fingered diodes(red or blue), blurring angle sensitivity. FIG. 28 e illustrates howembedding interleaved diodes in a well causes deeper photons to beabsorbed in or near the well's depletion regions, (violet) preventingblurring while still detecting all photons for intensity measures. FIG.28 f shows an alternate interleaved design with three wider diodes tocapture three angles.

Such diodes can be made using the source/drain diffusions of a CMOSprocess, which are typically very shallow. Since these depths (also <0.5μm) are significantly shallower than the penetration depth of visiblelight, much of the light will be absorbed at depths greater than that ofthe photodiodes themselves. This implies both a reduced quantumefficiency as some photo-generated carriers diffuse away or recombineand, that many (likely the majority) of the photo-generated carrierswill diffuse at least 0.5 μm before being absorbed by a photodiode. As aresult, photons absorbed below the photodiodes will generate carrierswith a roughly equal chance of being absorbed by either photodiode,badly blurring their ability to detect shifts in the Talbot pattern (seeFIG. 28 d). This is a known problem with fine pitch photojunctions, andhas limited resolution in other high-resolution imaging systems.

To avoid this penetration/diffusion problem, the interleaved, “fingereddiodes” can be placed in a deeper, more lightly doped well. These wellsare available in most modern CMOS processes with both n- and p-doping(FIG. 28 e). By providing a shallow boundary to the silicon below thefingered photodiodes, this well would absorb most of the carriersgenerated by deeply penetrating photons, reducing blurring. By measuringthe current generated by these deep photons overall intensity could bemeasured with high quantum efficiency, even as the interleaved diodesextract incident angle information. Simple simulations indicate that fora reasonable CMOS process (at the 130 nm or 180 nm process node),approximately 30% of green light would contribute to angle-sensitivephotocurrents, providing similar modulation depth when compared tostacked amplitude gratings. The remaining light would provide additionalintensity information. The ratio of well current to total finger currentin such a structure would also provide information about the wavelengthof the light detected since the average penetration depth of a photondepends upon its wavelength.

A second challenge with interleaved diodes is that the pitch theyrequire is very close to the limit of what is possible in reasonablypriced (i.e., 180 nm) CMOS. One approach to reduce this problem whilefurther increasing density is shown in FIG. 28 f. Simulations show thatthree distinct sets of fingered diodes, at a pitch of ⅔ the gratingpitch, extract three periodic, angle sensitive signals split by 120degrees (as compared to the 90 degree splits described hereinabove forthe stacked amplitude grating ASPs). These three independentmeasurements are sufficient to extract everything about a simple Talbotself-image. Specifically, these three numbers map, after transformationto the total light intensity, modulation depth of the self image(reflects how “blurry” the incident angle is), and the angle itself.Thus such a structure can provide extremely dense ASPs.

Both phase gratings and interleaved photodiodes can be independentlydeployed as improvements to stacked grating ASPs. The greatest benefitwill result when combining these approaches to produce a sensorstructure with an overall quantum efficiency and density equivalent to atypical angle insensitive pixel while simultaneously capturingadditional information about the light field.

The accuracy of an ASP array in localizing light sources in 3D space isinversely proportional to the angular gain of an ASP. This gain isproportional to z/d, where d is the grating pitch and z is the verticalseparation between grating and analyzer structure. All of thesestructures have made use of the ½ Talbot depth, where, at the optimalz=d²/λ, the angular gain is simply d/λ and is limited by the availablelayer stack, metal dimensions, and requirement that d>λ (required toobserve the Talbot effect). However, one is not confined to using the ½Talbot depth. Simulations show that the Talbot effect producesself-images at a variety of depths beyond the ½ Talbot depth. As thelateral shift of the Talbot effect is Δx=z tan(α), increasing z bychoosing a deeper Talbot self-image results in a greater lateral shiftfor a given incident angle, and therefore greater angular gain.

An alternative approach to achieving greater angular gain would be touse a higher periodicity self-image. For example, the phase gratingsimulation of FIG. 23 shows Talbot patterns with higher (particularly,double) periodicity than the generating grating at depths above andbelow the ½ Talbot depth. If we place interleaved diodes or analyzergratings at the depth and with the same pitch as these fractionalpatterns, the same lateral shift will correspond with a stronger changein response. This also corresponds to higher angular gain.

For a fixed depth between the initial grating and the sensor or analyzergrating, the pitch of a phase grating will be twice that of an analyzergrating. Therefore the default angular sensitivity (b) of phase gratings(computed by b=depth divided by pitch times 2n) would be half that ofsimilar amplitude gratings, barring further intervention. However, it ispossible to exploit higher-order Talbot patterns such as the ¼ orderpatterns, as shown in FIG. 29. FIG. 29 a shows a cross section of adevice engineered to use the 10¼ order Talbot depth, where the period ofintensity is twice the pitch at the diffractive layer. FIG. 29 b showsthat there are many higher-order Talbot depths available; all depthswith higher spatial periodicity than the phase grating will exhibithigh-order Talbot patterns (and thus greater angular sensitivity). FIG.29 c shows the simulated angular sensitivity of this device; it has a bof 66.

Since phase gratings are used to induce a phase shift, whereas materialsof different indices of refraction contribute to different optical pathlengths, one would assume that the depth of phase gratings should beproportional to the wavelength in use. When polychromatic light isobserved, the depths of the phase gratings will not be tunedappropriately for all wavelengths without intervention. It is possibleto mitigate this effect by using a dispersive substance such that oversome range of wavelengths the difference in refractive indices of Si0₂and the superstrate are somewhat proportional to wavelength. Forexample, if a plastic with similar optical dispersion to polyacrylatewere patterned on top of the Si0₂, the ideal grating thickness becomesless sensitive to wavelength. As shown in FIG. 30, with a water coating,the range of appropriate phase grating heights is roughly proportionalto wavelength. With a dispersive medium (such as a polyacrylate layer),some of the wavelength dependence can be mitigated since the refractiveindices of the media composing the gratings are closer at lowerwavelengths than at high wavelengths.

Four variants of ASPs have been experimentally characterized and arelisted in Table I.

TABLE I FABRICATED ANGLE SENSITIVE PIXEL STRUCTURES ASP Top AnalyzerSignals/ type Grating grating Photodiode type pixel I AmplitudeAmplitude N-well/p-substrate 1 II Amplitude None Interleavedn+/p-substrate 2 III Phase Amplitude N-well/p-substrate 1 IV Phase NoneInterleaved n+/p-substrate 2

The angular response of the ASPs were measured under green (523 nm,spectral half width 20 nm) plane wave illumination over an 80° range ofincident angles. FIGS. 31 a-d show the measured four phase output foreach of the four ASP types. All four ASP structures trace the desiredquadrature angle response. In the case of ASPs with interleavedphotodiodes, complementary phase responses were measured from a single 8μm×8 μm pixel. Newer ASP designs, types II, III and IV, exhibitmodulation depths in the range 0.2-0.3 as compared to amplitude gratingASPs with peak modulation depths above 0.4. For interleaved diode basedASPs, this degradation could be due to stray carriers from the substrateregions between and below the interleaved diodes. Deviations from theideal phase step of π/2 likely result in poorer modulation of intensitypatterns generated by the phase gratings. Further characterization isrequired to establish the effect of variations in the grating step.

For each ASP structure, the dependence of output on grating pitch andposition was characterized. Measurements were made for ASPs with gratingpitch, d, ranging from 0.6 μm to 3.2 μm and four different verticalgrating separations allowed by the CMOS metal stack. The measured andsimulated variation of modulation depth, m, with grating pitch is shownin FIG. 32 for an amplitude grating (type I) ASP with analyzer gratingdepth, z=2.76 μm and a phase grating ASP (type III) with analyzergrating depth, z=2.33 μm. For a desired angular sensitivity of 12, wepredict a grating pitch, d=1.03 μm for the type I ASP and d=0.87 μm forthe type III ASP. These calculations are consistent with measuredresults.

The QE of all the ASPs were measured and normalized to ann-well-p-substrate diode without gratings. The measured quantumefficiency of the four variants of ASPs shown in FIG. 33 confirms thatthe efficiency loss caused by metal gratings can be recovered by acombination of phase gratings and interleaved diodes.

For many applications, temporal information of incident light as well asspatial and angular information is important. One example is LIDAR, inwhich the time delay between light pulse emission and reflected pulsedetection provides a measure of distance. Fluorescence imaging can alsobenefit from temporal information, which provides a means todiscriminate between stimulation photons and fluorescence photonswithout complex filter optics. Combining angle sensitivity with veryhigh frame rates (upwards of 100M frames/s) allows for these and otherapplications to take advantage of the additional information containedin incident angle.

Existing research has shown that stand-alone CMOS chips can perform thisclass of high speed temporal imaging. Since the ASP can be manufacturedin a standard CMOS process, and since angle sensitive behavior relies onpurely optical phenomena, we believe high speed CMOS imaging techniquescan be integrated into angle-sensitive pixels. For example, high speeddevices, such as single-photon avalanche diodes and advanced detectorsdesigned specifically for time resolved imaging as methods, may beemployed to achieve this integration. These time-resolved ASPs willgreatly enhance existing high speed image sensors.

Current research is aimed toward small arrays of ASPs which are capableof performing bioassays without a microscope. However, for real-worldimaging applications, small arrays are incapable of capturing the fullcomplexity of observed scenes. Large array of at least 200,000 ASPs mayallow for real-world imaging applications of our enhanced ASPs. Thishigh resolution will demonstrate the compatibility of ASPs withtraditional imaging applications, and provides a platform whichdemonstrates the richness of information gathered by an ASP array incomparison to a pure intensity imager.

To achieve the required density within a small, low cost chip, variousmethods for ASP miniaturization can be employed. For example, currentASPs have distinct, local gratings for each diffraction grating/analyzergrating stack. These finite gratings exhibit edge effects which degradethe Talbot self images and therefore require a minimum number ofrepeated periods for robust function. Initial simulations have shownthat different sensors can share the Talbot self images generated by asingle, shared grating without compromising performance. This shouldpermit drastic grating size reduction, since the required large gratingis now amortized among several smaller sensors. Combined withalternative detector circuits which reduce the space required forreading out measured data and interleaved diodes, we believe that thesize of existing ASPs can be reduced by an order of magnitude. This willresult in low-cost, high-resolution light field sensors which will findwide application.

Another embodiment of the invention is directed to a method forextracting incident light-angle information from a light source object.The ability to detect said angle information has applicationspertaining, but not limited to sensor networks and sensor arrays;direction and/or speed of motion detection of objects passing over asensor such as, for example, the direction and speed of a vehiclepassing over a sensor embedded in road pavement; detection/reception ofmultiple data streams from separate transmitters in a free-space opticalcommunication system; bio-medical applications such as, e.g., detectionof individual fluorescent cells in tissue containing such fluorescentcells, and others that would be appreciated by persons skilled in theart.

According to a non-limiting aspect, the method may be accomplishedentirely without the use of lenses, and performed with image sensors onthe physical scale of silicon integrated circuits. For example, a sensorarray as described herein above may be positioned adjacent a piece oftissue containing fluorescent cells. Each cell would generate lightwhose incident angles (into the sensor) would indicate the cell'slocation in three-dimensional space. By triangulating back from theangles detected by each sensor in the array, as schematicallyillustrated in FIG. 1 b, the location of individual fluorescent cellscould be detected, as well as changes in their fluorescenceindependently from the other fluorescent cells.

According to a non-limiting aspect, many image sensor arrays could bedeployed as part of a larger array of tissue culture chambers. Eachindividual image sensor would provide monitoring of its respectivesample, providing high-throughput analysis of many samples at once.

According to a non-limiting, alternative aspect, the method may beaccomplished using an imaging sensor in combination with a lens system,which may be advantageous to image more distant objects. For example, asshown in FIG. 20 a, an object 1402 at the focal plane of the lens system1405 will appear completely in focus on the surface 1406 of the sensorarray 1407, and will appear to have an even distribution of angles ofincidence 1409. In this case the array acts like a normal CMOS imager.Objects more distant than the focal plane of the lens system willgenerate blurred images on the array, but the blurring will show avariable set of incident angles that converge on the focal plane of theobject, as shown in FIG. 20 b. Objects closer than the focal depth ofthe lens system will also appear blurred, but with a divergent set ofincident angles, indicating a focal depth behind the array, as shown inFIG. 20 c. Thus the imager can extract useful information that can beused to describe the location of objects both closer and farther awaythan the optical focal plane of the lens system. In other words, theimager, by detecting incident angle, can extract information about anobject region that is thicker than the normal depth of focus associatedwith a given lens system. Thus this information may be used, e.g., toreconstruct the three-dimensional structure of a scene or, tocomputationally refocus the image to different focal depths after theimage has been captured. The data from such an imager may be used tosimultaneously refocus different parts of the image to different depths.And although a single light-emitting object will generate a sensorresponse that maps to a single incident angle, multiple light sourceswill result in a linear superposition of responses, each of whichdepends on the incident angle of each source.

Since the diffraction grating approach to light field imaging describedherein is sensitive to the wavelength of light, a given pixel designwill work effectively for a limited range of wavelengths. Forapplications where this wavelength is known in advance, such as influorescent microscopy or communication systems using known LEDs, thegratings can be designed appropriately as known in the art. In imagingand/or angle detection applications using white light, a color filter1430 may be required in conjunction with the chip to limit the incidentlight wavelengths to the appropriate range. Modern imager processestypically include such color filters; thus they could be incorporatedinto the basic design. Furthermore, since such processes typicallyinclude multiple color filter layers (typically red, green and blue),using these filters in conjunction with three separately tuned sets ofgratings (tuned for these different colors) would permit light-fieldimaging in color.

It is possible to extract three-dimensional information about the sourceobject that is generating the responses of the diodes. Light originatingat any given point in space visible to the imager will generate a uniqueset of responses in the detectors in the imager. In particular, a pointat location (x,y,z) will illuminate a point (x_(s), y_(s)) on the imager(defining the plane of the imager as z_(s)=0) with an intensityproportional to

$B_{s} = \frac{B}{\left( {x - x_{s}} \right)^{2} + \left( {y - y_{s}} \right)^{2} + z^{2}}$

with incident angles:

$\theta_{x} = {\cos^{- 1}\left( \frac{z}{\sqrt{\left( {x - x_{s}} \right)^{2} + z^{2}}} \right)}$$\theta_{y} = {\cos^{- 1}\left( \frac{z}{\sqrt{\left( {y - y_{s}} \right)^{2} + z^{2}}} \right)}$

where θ_(x) is the azimuth and θ_(y) is the altitude. The resultingillumination of individual diodes in that region will follow theequations above, such that, for example, the “zero degree, horizontal”diode will see a brightness of

$D_{0\; H} = {\frac{I_{o}}{\left( {x - x_{s}} \right)^{2} + \left( {y - y_{s}} \right)^{2} + z^{2}}\frac{\left( {1 + {\cos \left( {k\; \theta_{x}} \right)}} \right)}{2}}$$D_{0\; H} = {\frac{I_{o}}{\left( {x - x_{s}} \right)^{2} + \left( {y - y_{s}} \right)^{2} + z^{2}}\frac{\left( {1 + {\cos\left( {k\; {\cos^{- 1}\left( \frac{z}{\sqrt{\left( {x - x_{s}} \right)^{2} + z^{2}}} \right)}} \right)}} \right)}{2}}$

and so on, such that any given diode's response to illumination from agiven point in space can be calculated. That is, one can define aresponse r(x_(s), y_(s), α) (where x_(s) and y_(s) are as above, and αis an index from 1 to 8 representing the phase associated with thediode) to any given stimulus s(x,y,z). This can be rewritten in vectorform by giving each diode an index i, (such that each combination x_(s),y_(s), α has a unique i: for example, for an N×N array with 8 possibleangles, i=α+8*x_(s)+N*8*y_(s), such that i ranges from 1 to 8N²). Onecan then define an overall response in the array r to a given stimuluspoint s(x,y,z), where each entry in r is defined as above. If one thencalculates this vector for every stimulus location in a volume withdimensions X, Y and Z, (defined as integer multiples of the resolutionone wants to image the volume with), one can define a single index, j,for these points in the volume, defined similarly to above. Thus one candefine any pattern of light sources in the volume of interest by asecond vector s. Since light at each point in s will cause a response ineach diode in r, and the effects of each of these light sources will addlinearly, one can define a matrix A, where each entry a(i,j) is definedas the response of diode i to a unit light stimulus at point j. Itfollows that

r=As,

where r is now a vector of diode responses that captures the totalresponse of the array to a given stimulus pattern in three dimensions.

It is noted that A is not a square matrix, since r has a total of 8N²entries, whereas s has XYZ entries. In most cases where a reasonablyhigh resolution is called for, one can assume that roughly, X=Y=N, and Zis on the order of N as well. Thus one can typically assume that s hasmany more entries than r (on the order of N/8 times as many).

In order to find s (i.e., the three-dimensional structure of the objectbeing imaged), matrix A is inverted by:

s=A ⁻¹ r.

However, such an inversion is not mathematically possible since A is notsquare matrix, but is “taller” than it is “wide”. There will not beenough information in a planar imager to distinguish all possiblestructures in a volume being imaged since there are simply more unknowns(the entries in s) than there are equations to define them (the entriesin r). Thus a solution to the problem requires that additionalconstraints be imposed. Two non-limiting examples are discussed below.

Example 1 Using Light Field Data for Refocusing and Range-Finding

One way to constrain the problem described above is to assume that s isthe result of visual information at a particular focal depth, andfinding the best estimate for what the image should look like if thedetected light originated only in that plane. s now describes a planewith dimensions X×Y, such that s has X*Y=N² entries. Since this is nowactually less than the size of r, A is still not square, but is now“wider” than it is “tall” such that the problem is now-over defined. Byusing a pseudo-inverse, usually defined for over constrained systems as(A^(T)A)⁻¹A^(T), one can extract a best fit for s given r. If there is areasonably good fit (as there will be if the focal plane chosen is thecorrect one), then this approach will yield a good approximation of theactual object scene. In particular, if the approximation of s is:

s′=p _(inv)(A)r,

where p_(inv)(A) is the pseudo-inverse of A, and given that A was chosensuch that

r=As,

then by the definition of the pseudo-inverse, the total error|s′−s| is minimized.

This approach can also be applied to data from a normal imager. However,since the mapping between s and r is different, the matrix A will bedifferent, as will as its pseudo-inverse, p_(inv)(A). In particular, thedegree to which a pseudo-inverse provides a useful result depends uponthe singular values of A (similar to eigenvalues in a square matrix).Moreover, the larger the singular values of a matrix, the less sensitivethe inversion process is to small errors and noise. The type of arraydescribed here, when used to compute the stimulus s for a given focalplane offset from the plane of the imager, provides a significantlyricher description resulting in a matrix with larger singular values. Anexample of this is shown in FIGS. 15 a, b for two 16×16 arrays, onewhere each pixel simply detects light, and the other where each pixelcontains two fingered diodes and a metal grating, such that sets of fourpixels form a cell, as shown in FIG. 5 c. For a focal plane four pixelwidths from the imager plane, the grating based design generates aconversion matrix A whose singular values are consistently larger, by asmuch as a factor of 100 than those for a normal imager. As a result,calculating the pseudo-inverse for a grating-based imager yields a moreaccurate, lower noise result than with a normal imager.

A second result of this approach is that one can calculate an error termbased upon using the pseudo inverse. In particular, calculating anestimate of r as:

r′=As

lets one then find an error term associate with this estimate:

err=|r′−r|.

If the estimate s′ accurately leads to the entries in r, then this errorterm will be small. This will be true if the source of the image was infact at the focal depth used when estimating A. On the other hand, ifthe image originates at some other focal depth, then the estimate willlikely be wrong, and the error term will be larger. Simulations confirmthis (again for the 16×16 array), with this error term minimized whenthe actual focal depth is chosen. This is distinct from the case of anormal imager where this error increases monotonically with estimatedfocal depth regardless of reality (see FIG. 15 b). Since this error termcan be calculated using only A and r without knowing s a priori, itshould be possible to use this error term to recognize the “correct”focal depth when refocusing the image. This information alone can beused for range-finding in a light field independent of the details ofthe object(s) being imaged.

The method described herein above need not be applied to the entireimage, but can be applied to subsections of the image such that they canbe refocused independently and/or their range found, leading to betteroverall focus and/or a range map across the image.

Example II Using Light Field Data for Extraction of Sparse FluorescentSources

Another exemplary application pertains to imaging the three dimensionalstructure of fluorescing cells in tissue. Since the goal of such anapplication would be to be able to independently quantify thefluorescence of multiple cells at different focal planes, refocusing isnot an appropriate approach. However, if one adds the two additionalconstraints: i) that all entries in s must be strictly positive (thereis no such thing as negative fluorescence), and ii) that the fluorescentsources are relatively sparsely distributed in the volume being imaged,one may assume that the number of fluorescent cells is smaller than N²,the number of pixels. If this holds true, then one can find each ofthese sources iteratively and in order of brightness as follows:

a) correlate r with the expected response to each possible entry in sas:

c=rA ^(T);

b) find the index, j, of s that correlates best with r (index of themaximum entry in c);

c) estimate the maximum value of s at this index that would yield aresponse r′ such that r(i)′<r(i) for all indices i. This implies thatr(i)−gA(j,i)>0, where A(i,j) is the i^(th) entry of the i^(th) column.Therefore g=min(r(i)/A(j,i)) across all values of i;

d) reduce by a factor λ, where 0<λ<1 and add to the existing estimate ofs

s′(j)=s′(j)+λg;

e) update residual value of r:

r=r−As′;

f) repeat steps (a-e).

Each iteration of this algorithm finds the most likely dominant pointsource of the light field seen in r, includes that source in theestimate of the stimulus, s′, then removes that portion of the effectfrom r permitting the algorithm to find the next most dominant source. λis chosen to be <1 so that no entry of r is driven to zero prematurely.In a simulation that provided reasonable results, λ=0.5.

The various embodiments described herein are compatible with a varietyof integrated light detectors/sensors including, without limitation,reverse-biased p-n junction diodes, forward bias diodes (i.e.,photovoltaics), p-i-n diodes, charge-coupled devices (CCDs),single-photon avalanche diodes, or pairs of interleaved N+/p-substratediffusion diodes.

The use of the terms “a” and “an” and “the” and similar references inthe context of describing the invention (especially in the context ofthe following claims) are to be construed to cover both the singular andthe plural, unless otherwise indicated herein or clearly contradicted bycontext. The terms “comprising,” “having,” “including,” and “containing”are to be construed as open-ended terms (i.e., meaning “including, butnot limited to,”) unless otherwise noted. The term “connected” is to beconstrued as partly or wholly contained within, attached to, or joinedtogether, even if there is something intervening.

The recitation of ranges of values herein are merely intended to serveas a shorthand method of referring individually to each separate valuefalling within the range, unless otherwise indicated herein, and eachseparate value is incorporated into the specification as if it wereindividually recited herein.

All methods described herein can be performed in any suitable orderunless otherwise indicated herein or otherwise clearly contradicted bycontext. The use of any and all examples, or exemplary language (e.g.,“such as”) provided herein, is intended merely to better illuminateembodiments of the invention and does not impose a limitation on thescope of the invention unless otherwise claimed.

No language in the specification should be construed as indicating anynon-claimed element as essential to the practice of the invention.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the present inventionwithout departing from the spirit and scope of the invention. There isno intention to limit the invention to the specific form or formsdisclosed, but on the contrary, the intention is to cover allmodifications, alternative constructions, and equivalents falling withinthe spirit and scope of the invention, as defined in the appendedclaims. Thus, it is intended that the present invention cover themodifications and variations of this invention provided they come withinthe scope of the appended claims and their equivalents.

1. An angle-sensitive pixel (ASP) device, comprising: a substrate; a phase grating having a period, p₁, disposed in the substrate, wherein the phase grating is characterized by a periodic variation of refractive index in a direction transverse to an incident light field and forms a periodic intensity pattern in a Talbot plane below the phase grating; an amplitude transmission analyzer grating having a period, p₂ disposed in the Talbot plane; and a sensor disposed at a selected distance below the analyzer grating.
 2. The ASP device of claim 1, further comprising a dispersive material patterned over the phase grating.
 3. The ASP device of claim 1, wherein the phase grating comprises silicon dioxide.
 4. The ASP device of claim 1, wherein the sensor comprises at least one pair of periodic, interleaved N+/p-substrate diffusion diodes.
 5. The device of claim 1, wherein p₂=p₁.
 6. The device of claim 1, wherein the analyzer grating is disposed at a second selected Talbot distance z_(T2)=(m₂/n₂)(2p₁ ²/λ), where m, n are positive integers and p₁ is equal to or greater than λ.
 7. The device of claim 1, wherein the sensor is disposed at a first selected Talbot distance z_(T1)=(m₁/n₁)(2p₁ ²/λ), where m, n are positive integers and p₁ is equal to or greater than λ.
 8. The device of claim 1, wherein the device is an integrated CMOS semiconductor structure.
 9. An angle-sensitive pixel (ASP) device, comprising: a substrate; a phase grating having a period, p₁, disposed in the substrate, wherein the phase grating is characterized by a periodic variation of refractive index in a direction transverse to an incident light field and forms a periodic intensity pattern in a Talbot plane below the phase grating; and a sensor disposed in the Talbot plane.
 10. The ASP device of claim 9, further comprising a dispersive material patterned over the phase grating.
 11. The ASP device of claim 9, wherein the phase grating comprises silicon dioxide.
 12. The ASP device of claim 9, wherein the sensor comprises at least one pair of interleaved N+/p-substrate diffusion diodes.
 13. The device of claim 9, wherein the sensor is disposed at a selected Talbot distance z_(T1)=(m₁/n₁)(2p₁ ²/λ), where m, n are positive integers and p₁ is equal to or greater than λ.
 14. The device of claim 1, wherein the device is an integrated CMOS semiconductor structure.
 15. A light field imaging device comprising a two-dimensional, M×N array of ASP devices according to claim 1 or 9, where M, N are integers equal to or greater than one.
 16. A method for making a phase-grating-based angle sensitive pixel (ASP), comprising the steps of: a) providing a standard CMOS stack including a plurality of patterned metal layers embedded in silicon dioxide, wherein the plurality of patterned metal layers characterize top and bottom regions of a phase grating of the phase grating-based ASP; b) etching the silicon dioxide and stopping on at least one of the patterned metal layers to define at least a high part of the phase grating of the phase grating-based ASP; c) either i) etching the silicon dioxide and stopping on at least two of the patterned metal layers to etch out the high part and a low part of the phase grating of the phase grating-based ASP, or ii) use a timed oxide etch to etch out a low part of the phase grating of the phase grating-based ASP; and d) etching the patterned metal layers so as to create a patterned interface between the silicon dioxide and an adjacent material, representing the phase grating of the phase grating-based ASP;
 17. The method of claim 16, further comprising choosing a pair of the metal layers to control a height of the phase grating.
 18. The method of claim 16, wherein step (ii) is a deep reactive ion etch (DRIE).
 19. The method of claim 16, wherein step (d) is either a wet, dry, or wet and dry etch. 